On Evaluation of Ensemble Forecasts Calibration Using the Concept of Data Depth

Various generalizations of the univariate rank histogram have been proposed to inspect the reliability of an ensemble forecast or analysis in multidimensional spaces. Multivariate rank histograms provide insightful information about the misspecification of genuinely multivariate features such as the correlation between various variables in a multivariate ensemble. However, the interpretation of patterns in a multivariate rank histogram should be handled with care. The purpose of this paper is to focus on multivariate rank histograms designed based on the concept of data depth and outline some important considerations that should be accounted for when using such multivariate rank histograms. In order to generate correct multivariate rank histograms using the concept of data depth, the datatype of the ensemble should be taken into account to define a proper pre-ranking function. We demonstrate how and why some pre-ranking functions might not be suitable for multivariate or vector-valued ensembles and propose pre-ranking functions based on the concept of simplicial depth that are applicable to both multivariate points and vector-valued ensembles. In addition, there exists an inherent identifiability issue associated with center-outward pre-ranking functions used to generate multivariate rank histograms. This problem can be alleviated by complementing the multivariate rank histogram with other well-known multivariate statistical inference tools based on rank statistics such as the depth-versus-depth (DD) plot. Using a synthetic example, we show that the DD-plot is less sensitive to sample size compared to multivariate rank histograms.